Explore the complete analysis inside the Version Insights

Performance Analysis Summary: PR #612

Overview

This PR introduces Q4_K block repacking optimization for AArch64 architecture using NEON SIMD instructions. The implementation adds a new 4x8 interleaving pattern (block_q4_Kx4) with three new kernel functions: ggml_quantize_mat_q8_K_4x8, ggml_gemv_q4_K_4x8_q8_K, and ggml_gemm_q4_K_4x8_q8_K. Changes span 4 files with 936 additions and 17 deletions, primarily affecting the GGML CPU backend's quantization and matrix multiplication paths.

Key Findings

Performance-Critical Function Changes

Matrix Quantization (ggml_quantize_mat_q8_K_4x8)

• Throughput improved by 2085 ns (from 2150 ns to 65 ns)
• Response time increased by 79 ns (from 2332 ns to 2411 ns)
• The function now delegates quantization work to specialized NEON kernels, reducing self-execution time while slightly increasing total path time due to function call overhead

Matrix-Vector Multiply (ggml_gemv_q4_K_4x8_q8_K)

• New NEON-optimized implementation using DOTPROD instructions
• Processes 4 columns simultaneously with 8-byte interleaving
• Replaces generic scalar implementation with vectorized operations

Matrix-Matrix Multiply (ggml_gemm_q4_K_4x8_q8_K)

• New NEON-optimized implementation using MATMUL_INT8 extension
• Processes 16 rows × 4 columns simultaneously
• Utilizes vmmlaq_s32 intrinsic for 2×8 matrix multiply operations

Impact on Inference Performance

The modified functions operate at the quantization and low-level matrix operation layer, not the tokenization or inference orchestration layer. Functions like llama_decode, llama_encode, and llama_tokenize remain unchanged in this PR. Therefore, tokens per second metrics are not directly impacted by these changes. The optimizations affect the underlying computational kernels that these higher-level functions call, potentially improving their execution efficiency on AArch64 hardware with NEON support, but the inference API layer itself is unmodified.

Power Consumption Analysis

Binary: libggml-cpu.so

• Power consumption increased by 2517 nJ (from 119985 nJ to 122502 nJ, +2.10%)
• The increase stems from cumulative overhead in parameter-setting utility functions that are called frequently during tensor operations
• All other binaries (libllama.so, libggml-base.so, llama-run, etc.) show no measurable power consumption change

The power increase is concentrated in the CPU backend library where the new repacking logic executes. The 2.10% increase represents the energy cost of pre-processing (repacking) operations that enable faster runtime execution on NEON-capable hardware.